Image processing apparatus, image processing method, and storage medium

ABSTRACT

An image processing apparatus is provided with a spatial information calculation unit for calculating spatial information of a subject, which is the information of an area in which the subject in an image is predicted to be present, a first area setting unit for setting a first area in the image based on the spatial information, a second area setting unit for setting a second area outside the first area, a first feature amount calculation unit for calculating a first feature amount of the first area, a second feature amount calculation unit for calculating a second feature amount of the second area, the second feature amount being a feature amount of the same type as the first feature amount, and an saliency calculation unit for calculating a degree of visual saliency of the subject.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a technique for detecting a visuallyconspicuous area in a captured image.

2. Description of the Related Art

In recent years, a technique for detecting, among subjects that arepresent in an image, a subject that is likely to be conspicuous to humanvision has been proposed. For example, in Japanese Patent Laid-Open No.2012-123631, a visually salient area is detected based on a differencein feature amount between a training data extraction area with a smallradius and a verification data extraction area with a large radius. Thatis, it is determined that the larger the difference in feature amountbetween the training data extraction area and the verification dataextraction area surrounding the training data extraction area is, thehigher the degree of visual saliency (hereinafter, referred to as thedegree of visual saliency) is.

In the above-described Japanese Patent Laid-Open No. 2012-123631, thesizes of the training data extraction area and the verification dataextraction area are determined based on learning data of a subject, andthe degree of visual saliency at an arbitrary point in the screen iscalculated. However, there is a problem in that the sizes of thetraining data extraction area and the verification data extraction areacannot be determined in the case of a subject whose learning data doesnot exist, and the accuracy of the degree of visual saliencydeteriorates. There is also a problem in that, for example, if aplurality of areas with different sizes are set and degrees of visualsaliency at arbitrary points are calculated, the processing amount willbe huge.

SUMMARY OF THE INVENTION

The present invention has been made in light of the above-describedproblems, so that a highly accurate degree of visual saliency iscalculated and an area of interest is detected with a small processingamount.

According to the first of the present invention, there is provided animage processing apparatus comprising: a spatial information calculationunit configured to calculate spatial information of a subject, thespatial information being information of an area in which the subject inan image is predicted to be present; a first area setting unitconfigured to set a first area in the image based on the spatialinformation; a second area setting unit configured to set a second areaoutside the first area; a first feature amount calculation unitconfigured to calculate a first feature amount of the first area; asecond feature amount calculation unit configured to calculate a secondfeature amount of the second area, the second feature amount being afeature amount of the same type as the first feature amount; and ansaliency calculation unit configured to calculate a degree of visualsaliency of the subject based on a difference between the first featureamount and the second feature amount.

According to the second aspect of the present invention, there isprovided an image processing method comprising: calculating spatialinformation of a subject, the spatial information being information ofan area in which a subject in an image is predicted to be present;setting a first area in the image based on the spatial information;setting a second area outside the first area; calculating a firstfeature amount of the first area; calculating a second feature amount ofthe second area, the second feature amount being a feature amount of thesame type as the first feature amount; and calculating a degree ofvisual saliency of the subject based on a difference between the firstfeature amount and the second feature amount.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing the configuration of a firstembodiment in which an image processing apparatus according to thepresent invention is applied to an image capturing apparatus.

FIG. 2 is a diagram showing the configuration of a spatial informationcalculation unit in a first embodiment.

FIG. 3 is a flowchart showing the processing of the spatial informationcalculation unit in the first embodiment.

FIGS. 4A to 4C are diagrams showing focus area information.

FIG. 5 is a diagram showing the configuration of an area-of-interestdetection unit in the first embodiment.

FIG. 6 is a flowchart showing the processing of the area-of-interestdetection unit in the first embodiment.

FIGS. 7A and 7B are diagrams for describing area information.

FIGS. 8A to 8C are diagrams for describing degrees of visual saliency inthe first embodiment.

FIG. 9 is a diagram showing the configuration of a spatial informationcalculation unit in a second embodiment.

FIG. 10 is a flowchart showing the processing of the spatial informationcalculation unit in the second embodiment.

FIGS. 11A to 11C are diagrams showing moving area information.

FIGS. 12A and 12B are diagrams for describing degrees of visual saliencyin the second embodiment.

FIG. 13 is a diagram showing the configuration of a spatial informationcalculation unit in a third embodiment.

FIG. 14 is a flowchart showing the processing of the spatial informationcalculation unit in the third embodiment.

FIGS. 15A and 15B are diagrams showing face area information.

FIGS. 16A and 16B are diagrams for describing degrees of visual saliencyin the third embodiment.

DESCRIPTION OF THE EMBODIMENTS

Embodiments of the present invention will be described below in detailwith reference to the attached drawings.

First Embodiment

FIG. 1 is a block diagram showing the configuration of the firstembodiment in which the image processing apparatus according to thepresent invention is applied to an image capturing apparatus. An imagecapturing apparatus 100 in the present embodiment detects an area ofinterest based on spatial information of a subject determined inaccordance with focus area information. The configuration of the imagecapturing apparatus of the present embodiment will be described belowwith reference to FIG. 1.

In FIG. 1, reference numeral 101 indicates a lens group including a zoomlens and a focusing lens, reference numeral 102 indicates a shutterprovided with a diaphragm function, and reference numeral 103 indicatesan image capturing unit constituted by a CCD, a CMOS element, or thelike that converts an optical image into an electrical signal. Referencenumeral 104 indicates an A/D converter that converts an analog signaloutput by the image capturing unit 103 into a digital signal, andreference numeral 105 indicates an AF sensor constituted by a CCD, aCMOS element, or the like that converts an optical image into anelectrical signal for AF control. Reference numeral 106 indicates an AFA/D converter that converts an analog signal output by the AF sensor 105into a digital signal. Reference numeral 107 indicates an imageprocessing unit that performs various types of image processing such aswhite balance processing or γ processing, on image data output from theA/D converter 104. Reference numeral 108 indicates an image memory,reference numeral 109 indicates a memory control unit that controls theimage memory 108, reference numeral 110 indicates a D/A converter thatconverts an input digital signal into an analog signal, referencenumeral 111 indicates a display unit such as an LCD, and referencenumeral 112 indicates a codec unit that performs compression coding anddecoding of the image data.

Reference numeral 113 indicates a storage medium such as a memory cardor a hard disk that stores the image data. Reference numeral 114indicates a storage I/F, which is an interface to the storage medium113. Reference numeral 115 indicates a spatial information calculationunit that calculates the spatial information indicating the position orthe size of a subject that is present in the image data. Referencenumeral 116 indicates an area-of-interest detection unit that detectsthe area of interest in the image data.

Reference numeral 50 indicates a system control unit for controlling theoverall system of the image capturing apparatus 100. Reference numeral120 indicates an operation unit for inputting various types of operationinstructions from a user, reference numeral 121 indicates a power supplyswitch, and reference numeral 122 indicates a power supply unit.Reference numeral 123 indicates an electrically erasable and recordablenonvolatile memory, for which an EEPROM or the like is used, forexample. Reference numeral 124 indicates a system timer for measuringtime used for various types of control or the time of an integratedclock, and reference numeral 125 indicates a system memory in whichconstants and variables for the operations of the system control unit50, a program read out from the nonvolatile memory 123, and the like areexpanded.

Next, a flow of basic processing during shooting in the image capturingapparatus 100 configured as described above will be described. The imagecapturing unit 103 photoelectrically converts light that is incidentthereon via the lens 101 and the shutter 102, and output the convertedlight as an input image signal to the A/D converter 104. The A/Dconverter 104 converts an analog image signal output from the imagecapturing unit 103 into a digital image signal, and outputs the digitalimage signal to the image processing unit 107. The AF sensor 105receives, using multiple pairs of line sensors, light that is incidentvia the lens 101 and the shutter 102, and outputs the light to the AFA/D converter 106. The AF A/D converter 106 converts an analog signaloutput from the AF sensor 105 into a digital signal, and outputs thedigital signal to the system control unit 50. The system control unit 50detects, based on the image signal output by a pair of the line sensors,a relative position shift amount from a subject in the splittingdirection of a luminous flux, and performs so-called phase difference AFcontrol.

The image processing unit 107 performs various types of image processingsuch as white balance processing or γ processing on the image data fromthe A/D converter 104 or image data read out from the memory controlunit 109. The image data output from the image processing unit 107 iswritten into the image memory 108 via the memory control unit 109.Moreover, the image processing unit 107 performs predetermined computingprocessing using the image data captured by the image capturing unit103, and the system control unit 50 performs exposure control or focusadjusting control based on the obtained computing result. Accordingly,AE (automatic exposure) processing, AF (autofocus) processing, and thelike are performed.

The image memory 108 stores the image data output from the imagecapturing unit 103 and image data to be displayed on the display unit111. Moreover, the D/A converter 110 converts the data for image displaystored in the image memory 108 into an analog signal and supplies theanalog signal to the display unit 111. The display unit 111 performsdisplay on a display device such as an LCD in accordance with the analogsignal from the D/A converter 110. The codec unit 112 performscompression coding on the image data stored in the image memory 108based on a standard such as JPEG or MPEG.

The spatial information calculation unit 115 calculates the spatialinformation indicating the position or the size of a subject present inan image. Then, based on the spatial information from the spatialinformation calculation unit 115, the area-of-interest detection unit116 detects the area of interest in which the subject in the image datais predicted to be present, and outputs area-of-interest information tothe system control unit 50. The system control unit 50 determines, basedon the area-of-interest information, an area (target area forprocessing) in which predetermined processing is preferentiallyperformed. The predetermined processing includes performing the AFcontrol so that the subject belonging to the area of interest is infocus in the case where multiple subjects are present in the image data,for example. The predetermined processing also includes performing theAF control so that the subject belonging to the area of interest hasproper brightness in the case where multiple subjects are present in theimage data. The spatial information calculation unit 115 and thearea-of-interest detection unit 116 will be described later.

Besides the above-described basic operations, the system control unit 50executes a program stored in the above-described nonvolatile memory 123so as to realize the processing of the present embodiment, which will bedescribed later. The program herein refers to a program for executingvarious flowcharts that will be described later in the presentembodiment. At this time, constants and variables for the operations ofthe system control unit 50, the program read out from the nonvolatilememory 123, and the like are expanded in the system memory 125. Theabove has described the configuration and the basic operations of theimage capturing apparatus 100.

Next, the spatial information calculation unit 115 and thearea-of-interest detection unit 116 will be described in detail. First,the spatial information calculation unit 115 will be described in detailwith reference to FIG. 2, FIG. 3 and FIGS. 4A to 4C.

FIG. 2 is a diagram showing the configuration of the spatial informationcalculation unit 115 in the present embodiment. The spatial informationcalculation unit 115 is constituted by a subject area detection unit201, a subject candidate area detection unit 202, and a subject spatialinformation generation unit 203.

FIG. 3 is a flowchart showing the processing of the spatial informationcalculation unit 115. The operations of the spatial informationcalculation unit 115 will be described below with reference to theflowchart in FIG. 3.

In step S301, the system control unit 50 outputs, to the subject areadetection unit 201, the information of a focus area in which adifference of phase is smaller than a predetermined value during thephase difference AF control (during focusing operation) (focusinformation calculation). Examples of the focus area information areshown in FIGS. 4A to 4C. FIG. 4A shows the image data captured by theimage capturing unit 103, and FIG. 4B and FIG. 4C show the focus areainformation created by the system control unit 50.

FIG. 4B shows the focus area information created by the system controlunit 50 based on the output signal of the AF sensor 105. In FIG. 4B, arectangle such as p100 or p110 indicates a single ranging point in theAF sensor 105, and it is indicated that a blackened ranging point suchas p110 is a non-focus point in which the phase difference is greaterthan or equal to the predetermined value and a whitened ranging pointsuch as p100 is a focus point in which the phase difference is smallerthan the predetermined value. That is, the example in FIG. 4B indicatesthat two birds in FIG. 4A are in focus.

In step S302, the subject area detection unit 201 sets the focus area asa subject area. Specifically, an area in which in-focus ranging pointsare grouped is set as a subject area. In FIG. 4B, in-focus points p100,p101, p102, and p103 are grouped and set as a subject area a200.Moreover, in-focus points p104, p105, p106, and p107 are grouped and setas a subject area a202.

In step S303, the subject area candidate detection unit 202 sets asubject candidate area. The subject candidate area refers to an area inwhich there is a possibility that a subject is present. If the rangingpoints are discrete as with the focus area information in FIG. 4B, forexample, a boundary position between a bird and a background (sky) inFIG. 4A cannot be precisely detected. In view of this, as the area inwhich there is a possibility that the subject is present, an areaextending to the non-focus ranging points that is present around thefocus area is assumed to be the subject candidate area. That is, asubject candidate area corresponding to the subject area a200 is assumedto be a201, and a subject candidate area corresponding to the subjectarea a202 is assumed to be a203.

In step S304, the subject spatial information generation unit 203converts subject area information calculated by the subject areadetection unit 201 and subject candidate area information calculated bythe subject candidate area detection unit 202 into coordinatescorresponding to image data, and outputs the data as spatialinformation.

In the above description, the AF sensor 105 having discrete rangingpoints as in FIG. 4B was described as an example, but the system controlunit 50 may calculate the focus area information using so-called imagecapturing plane phase difference AF in which AF control using a phasedifference method is performed using a pixel output signal of the imagecapturing unit 103. In this case, the image capturing unit 103 isconfigured to divide pixels under a single micro lens and separatelyreceive a portion of an exit pupil of a shooting optical system. Bydividing the pixels under the micro lens in this manner for all thepixels of the image capturing unit 103, it is possible to calculate aphase difference for each pixel, and the focus area information with ahigh resolution can be obtained.

FIG. 4C is a diagram showing an area (focus area information) in whichthe phase difference, which has been calculated for each pixel using theabove-described image capturing plane phase difference AF method, issmaller than the predetermined value. In FIG. 4C, a whitened area is thefocus area, and blackened area is a non-focus area. As shown in thedrawing, the focus area information calculated based on the imagecapturing plane phase difference AF method has a higher resolutioncompared with the focus area information calculated based on the phasedifference AF of the AF sensor 105 in FIG. 4B, making it possible tocapture the shape of the subject with high accuracy. In the case where asubject area can be detected with high accuracy in this manner, thesubject area candidate detection unit 202 sets a subject candidate areaas the same area as the subject area, and an area in which there is apossibility that a subject is present does not have to be separatelydetected.

Moreover, in the examples of FIG. 4B and FIG. 4C, the subject areadetection unit 201 detects, as the subject area, an area surrounding thesubject in the shape of a rectangle with respect to the focus areainformation, but detection of the subject area does not need to be inthe shape of a rectangle. For example, the shape may be a circle or atriangle, and the shape of the focus area itself in which the phasedifference is smaller than the predetermined value may be the subjectarea. That is, in the example of FIG. 4C, the focus area (whitened areain the shape of a bird) in an area a210 and the focus area (whitenedarea in the shape of a bird) in an area a211 may be the subject areas.The above has described spatial information calculation unit 115.

Next, the area-of-interest detection unit 116 will be described indetail with reference to FIG. 5, FIG. 6, FIGS. 7A and 7B and FIGS. 8A to8C. The area-of-interest detection unit 116 detects an area of interestin the image data based on the spatial information calculated by thespatial information calculation unit 115.

FIG. 5 is a diagram showing the configuration of the area-of-interestdetection unit 116. The area-of-interest detection unit 116 isconfigured by including a search area determination unit 501, a subjectinner area setting unit 502, a subject outer area setting unit 503, asubject inner area feature amount calculation unit 504, a subject outerarea feature amount calculating unit 505 and a visual saliencycalculation unit 506.

FIG. 6 is a flowchart showing the processing of the area-of-interestdetection unit 116. The operations of the area-of-interest detectionunit 116 will be described below with reference to the flowchart in FIG.6.

In step S601, the search area determination unit 501 determines a searcharea for calculating a degree of visual saliency based on the subjectcandidate area contained in the spatial information calculated by thespatial information calculation unit 115. FIG. 7A shows an example ofdetermining a specific search area. In FIG. 7A, as described withreference to FIG. 4, the subject candidate areas are the areas a201 anda203. The search area determination unit 501 determines these subjectcandidate areas a201 and a203 as the search areas. Note that in the casewhere the subject candidate area information is not contained in thespatial information, it is determined that the subject area is a searcharea.

In step S602, the subject inner area setting unit 502 sets, based on thesubject area information contained in the spatial information calculatedby the spatial information calculation unit 115 and the search areadetermined by the search area determination unit 501, a subject innerarea for calculating the degree of visual saliency. A specific exampleof setting the subject inner area is shown in FIG. 7A. In FIG. 7A, asdescribed with reference to FIGS. 4A to 4C, the subject areas are a200and a202. The subject inner area setting unit 502 sets these subjectareas a200 and a202 as the subject inner areas.

Moreover, the position of the subject area may be shifted in horizontaland perpendicular directions in the search area for each predeterminedpixel without changing the size of the subject area, so that multiplesubject inner areas are set, as with a200, a300 and a301 in FIG. 7B, forexample. Moreover, the size of the subject area may be increased in thesearch area for each predetermined pixel without changing the positionof the subject area, so that multiple subject inner areas may be set, aswith a202, a303 and a304 in FIG. 7B, for example. By performing theabove described operations, the degree of visual saliency, which will bedescribed later, can be calculated for an area in which there is apossibility that the subject is present and for the size of a subject.

In step S603, the subject outer area setting unit 503 sets, based on thesubject inner area set by the subject inner area setting unit 502, asubject outer area outside the subject inner area in order to calculatethe degree of visual saliency. A specific example of setting the subjectouter area is shown in FIG. 7A. In FIG. 7A, for example, the subjectinner area is a200. First, the subject outer area setting unit 503 setsan area a204 that is centered on this subject inner area a200 and islarger than the subject inner area a200, and sets, as the subject outerarea, the remaining area a204 excluding the subject inner area a200. Thesize of the area a204 is a predetermined number of times the length of aside or the area measurement of the subject inner area a200, forexample. Moreover, the subject outer area may be set to an area withouta focus point, for example. Moreover, in the case where the shape of thesubject inner area is circular or triangular, the shape of the subjectouter area may also be circular or triangular. Note that in the casewhere the subject inner area setting unit 502 sets multiple subjectinner areas, the subject outer area setting unit 503 sets a subjectouter area for each of the subject inner areas.

In step S604, the subject inner area feature amount calculation unit 504calculates a feature amount of the image data in the subject inner areaset by the subject inner area setting unit 502. FIG. 8A shows a subjectinner area for which the feature amount is to be calculated. In FIG. 8A,as described with reference to FIGS. 7A and 7B, the subject inner areasset by the subject inner area setting unit 502 are a200 and a202.Therefore, the subject inner area feature amount calculation unit 504calculates the feature amounts of these two areas.

In step S605, the subject outer area feature amount calculating unit 505calculates a feature amount of the image data in the subject outer areaset by the subject outer area setting unit 503. FIG. 8A shows subjectouter areas for which the feature amount is to be calculated. In FIG.8A, as described with reference to FIGS. 7A and 7B, the subject outerareas set by the subject outer area setting unit 503 are the remainingarea a204 excluding the area a200 and a remaining area a205 excludingthe area a202. Therefore, the subject outer area feature amountcalculating unit 505 calculates the feature amounts of these two areas.

Here, the feature amounts calculated by the subject inner area featureamount calculation unit 504 and the subject outer area feature amountcalculating unit 505 are feature amounts of the same type including atleast one of brightness, color, edge intensity, a brightness histogram,a color histogram and an edge intensity histogram in the area.

In step S606, the visual saliency calculation unit 506 calculates thedegree of visual saliency based on the difference between the subjectinner area feature amount calculated by the subject inner area featureamount calculation unit 504 and the subject outer area feature amountcalculated by the subject outer area feature amount calculating unit505. Specifically, the absolute difference of the feature amounts isevaluated so as to calculate the degree of visual saliency. For example,in the case where the feature amount involves brightness, the absolutedifference between the luminance average value of the subject inner areaand the luminance average value of the subject outer area serves as thedegree of visual saliency. That is, in the example of FIG. 8A, theabsolute difference between the luminance average value of, mainly, ablack bird in the subject inner area a200 and the luminance averagevalue of, mainly, the sky in the subject outer area serves as the degreeof visual saliency. Moreover, the absolute difference between theluminance average value of, mainly, a white bird in the subject innerarea a202 and the luminance average value of, mainly, the sky in thesubject outer area serves as the degree of visual saliency.

FIG. 8B is a diagram showing the degrees of visual saliency, in whichthe degree of visual saliency corresponding to the subject inner areaa200 (black bird) is indicated by an area a400, and the degree of visualsaliency corresponding to the subject inner area a202 (white bird) isindicated by an area a401. Moreover, FIG. 8B shows that the whiter thearea is, the larger the degree of visual saliency is, and the blackerthe area is, the smaller the degree of visual saliency is.

Regarding the degrees of visual saliency of the subject inner area a200(black bird) and the subject inner area a202 (white bird) calculated bythe above-described method, because the absolute difference of averageluminance of the black bird and the sky is larger than that of the whitebird and the sky, the degree of visual saliency of the black bird ishigher than that of the white bird as in FIG. 8B. If the size of thesubject outer area corresponding to the subject inner area a200 is toolarge or too small compared with the actual subject (black bird), theabsolute difference between the average luminance of the subject innerarea and the subject outer area will be small, and therefore, the degreeof visual saliency cannot be appropriately calculated. Therefore, asubject inner area suitable for the position or the size of an actualsubject needs to be set.

Moreover, for example, in the case where the feature amount involves abrightness histogram, the absolute difference per each bin between theluminance histogram of the subject inner area and the luminancehistogram of the subject outer area is calculated, and the integratedvalue thereof serves as the degree of visual saliency.

Moreover, the visual saliency calculation unit 506 calculates thedegrees of visual saliency for multiple subject inner areas and subjectouter areas set in the subject candidate area. In the case where themultiple subject inner areas are overlapped as in the example of FIG.8C, the degree of visual saliency of the overlapped subject inner areasis the average value of the multiple degrees of visual saliencycalculated for the overlapped subject inner areas. Moreover, the maximumdegree of visual saliency among the multiple degrees of visual saliencycalculated for the overlapped subject inner areas may serve as thedegree of visual saliency.

In step S607, in the case where the search for all of the search areasdetermined in step S601 is not finished, the processing of step S602 andonward is repeated.

Note that the visual saliency calculation unit 506 does not need tocalculate the degree of visual saliency in the case where the size ofthe subject inner area is smaller than a predetermined value. Moreover,in the case where the subject inner area is spaced away from the centerof a screen by a distance that is greater than or equal to apredetermined distance (positioned at the end of the screen), the visualsaliency calculation unit 506 does not need to calculate the degree ofvisual saliency. This is because the subject is less likely to be givenattention in the case where the size of the subject is small or in thecase where the subject is positioned at the end of the screen. The abovehas described the area-of-interest detection unit 116.

Because the degree of visual saliency calculated in this manner involvesa visually conspicuous area of interest, the degree of visual saliencyis used for priority setting for various processes. Examples of suchprocesses include performing the AF control preferentially on this areaof interest.

For example, in the case where multiple in-focus subjects move, thesystem control unit 50 performs the AF control based on the informationof the phase difference calculated for the AF control and theinformation of the area of interest detected by the area-of-interestdetection unit 116, so that the subject corresponding to the area ofinterest is preferentially in focus. Specifically, in the examples ofFIGS. 8A to 8C, in the case where the black bird and the white bird moveat the same time, the AF control is performed with priority given sothat the black bird having the larger degree of visual saliency is infocus. By performing the AF control preferentially on the area ofinterest in this manner, it is possible to cause the attention subject(area of interest) to be in focus without blurring the attention subject(area of interest).

Note that, in the above description, the subject area detection unit 201sets the subject area based on the area in which the phase difference issmaller than the predetermined value (focus area information), but amethod of setting a subject area is not limited thereto. For example,the system control unit 50 outputs information on multiple areas such asan area in which the phase difference is in a first range (front focusstate), an area in which the phase difference is in a second range(focus state), and an area in which the phase difference is in a thirdrange (rear focus state). Then, the subject area detection unit 201 maycalculate the subject area information based on each piece of areainformation.

The area-of-interest detection unit 116 detects an area of interestbased on this subject area information. If the area-of-interestdetection unit 116 detects a subject area in the front focus state asthe area of interest, the system control unit 50 performs the AF controlso that this non-focus subject area is preferentially in focus. It ispossible to focus on a non-focus area of interest by performing the AFcontrol as described above.

Second Embodiment

The image capturing apparatus 100 in accordance with the secondembodiment of the present invention detects an area of interest based onthe spatial information of a subject determined in accordance withmoving area information (moving body information). The second embodimentwill be described below. Note that components with the same referencesigns as those in the first embodiment perform operations and processingsimilar to those of the first embodiment, and therefore descriptionthereof is omitted.

The second embodiment is different from the first embodiment in theconfiguration and the operations of the spatial information calculationunit in FIG. 1. Because other configurations and operations are the sameas those of the first embodiment, description thereof is omitted. Thespatial information calculation unit 115 in the second embodiment willbe described in detail with reference to FIG. 9, FIG. 10 and FIGS. 11Ato 11C.

FIG. 9 is a diagram showing the configuration of the spatial informationcalculation unit 115 in the second embodiment. The spatial informationcalculation unit 115 is configured by including a moving area detectionunit 901, a subject area detection unit 902, and a subject spatialinformation generation unit 903.

FIG. 10 is a flowchart showing the processing of the spatial informationcalculation unit 115 in the second embodiment. The operations of thespatial information calculation unit 115 will be described below withreference to the flowchart in FIG. 10.

In step S1001, the image capturing unit 103 captures at least two imagesat different times as in FIG. 11A and FIG. 11B. FIG. 11A shows imagedata 1 of an Nth frame, and FIG. 11B shows image data 2 of an N+1thframe. In FIGS. 11A to 11C, an example of capturing a moving baseball isshown, and because the image data 1 and the image data 2 were capturedat different times, the moving subject (baseball) is present atdifferent positions.

In step S1002, the moving area detection unit (moving body informationcalculation unit) 901 calculates an inter-frame absolute difference foreach pixel between the two (multiple) images captured in step S1001.Moving area information in which an area with a large inter-frameabsolute difference serves as a moving area is then created as in FIG.11C. In FIG. 11C, the moving area with the large inter-frame absolutedifference is whitened. The position or the size of the subject can beunderstood by referring to such moving area information.

In step S1003, the subject area detection unit 902 sets the moving areaas a subject area. Specifically, as in FIG. 11C, subject areas a500,a501, a502, and a503 are set so as to surround the respective movingareas.

In step S1004, the subject spatial information generation unit 903outputs, as the spatial information, subject area information calculatedby the subject area detection unit 902.

The above has described the spatial information calculation unit 115 inthe second embodiment. The area-of-interest detection unit 116 of thepresent embodiment has a configuration and operations similar to thoseof the first embodiment, and therefore detailed description thereof isomitted.

FIGS. 12A and 12B are diagrams for describing the degree of visualsaliency calculated by the area-of-interest detection unit 116 in thesecond embodiment. The area-of-interest detection unit 116 in the secondembodiment has the same configuration and performs the same operationsas the first embodiment, so that subject inner areas and subject outerareas as in FIG. 12A are set based on the spatial information describedwith reference to FIGS. 11A to 11C.

Here, the subject inner areas are areas a600, a602, a604, and a606 inFIG. 12A. A subject outer area corresponding to the subject inner areaa600 is an area a601 in FIG. 12A excluding a600. Similarly, subjectouter areas corresponding to the subject inner areas a602, a604, anda606 are respectively areas a603, a605, and a607 excluding a602, a604,and a606.

The area-of-interest detection unit 116 then calculates the degree ofvisual saliency as in FIG. 12B based on the difference in feature amountbetween the subject inner area and the subject outer area. The degree ofvisual saliency of an area a700 will have a large value because thedifference in feature amount between the subject inner area and thesubject outer area is large. The degree of visual saliency of an areaa701 will have a moderate value because the difference in feature amountbetween the subject inner area and the subject outer area is moderate.The degrees of visual saliency of an area a702 and an area a703 willhave small values because the differences in feature amount between thesubject inner areas and the subject outer areas are small.

Because the degree of visual saliency calculated in this manner involvesa visually conspicuous area of interest, the degree of visual saliencyis used for priority setting for various processes. Examples of suchprocesses include performing the AF control preferentially on this areaof interest.

For example, in the case where multiple moving bodies (baseballs) arepresent as in FIG. 12A, the system control unit 50 performs the AFcontrol based on the information of a phase difference calculated forthe AF control and the information of an area of interest detected bythe area-of-interest detection unit 116, so that the subjectcorresponding to the area of interest is preferentially in focus.Specifically, in the examples of FIGS. 12A and 12B, the AF control ispreferentially performed on, out of an upper baseball and a lowerbaseball, the upper baseball whose degree of visual saliency is larger.By performing the AF control preferentially on the area of interest inthis manner, it is possible to cause the attention subject (area ofinterest) to be in focus without blurring the attention subject (area ofinterest).

Third Embodiment

The image capturing apparatus 100 according to the third embodiment ofthe present invention detects an area of interest based on spatialinformation of a subject determined in accordance with face areainformation. The third embodiment will be described below. Note thatcomponents with the same reference signs as those in the first andsecond embodiments perform operations and processing similar to those ofthe first and the second embodiments, and therefore description thereofis omitted.

The third embodiment is different from the first and second embodimentsin the configuration and the operations of the spatial informationcalculation unit in FIG. 1. Other configurations and the operations arethe same as those of the first embodiment, and therefore the descriptionthereof is omitted. The spatial information calculation unit 115 in thethird embodiment will be described in detail with reference to FIG. 13,FIG. 14 and FIGS. 15A and 15B.

FIG. 13 is a diagram showing the configuration of the spatialinformation calculation unit 115 in the third embodiment. The spatialinformation calculation unit 115 is configured by including a facedetection unit 1301, a subject area detection unit 1302, and a subjectspatial information generation unit 1303.

FIG. 14 is a flowchart showing the processing of the spatial informationcalculation unit 115 in the third embodiment. The operations of thespatial information calculation unit 115 in the third embodiment will bedescribed below with reference to the flowchart in FIG. 14.

In step S1401, the face detection unit 1301 detects a face area that ispresent in image data, and outputs face area information (face detectioninformation). The face detection unit 1301 will be described withreference to FIGS. 15A and 15B. Note that a known method such as anevaluation method of matching eyes and a nose with a learning pattern isused for detecting the face area, for example. The face detection unit1301, in which image data as in FIG. 15A is input, detects a face areathat is present in the image data, and outputs face area information asin FIG. 15B, for example. FIG. 15B showing the face area informationcorresponds to FIG. 15A showing the image data, and four face areas areindividually a800, a801, a802, and a803. The face detection unit 1301outputs such face area information to the subject area detection unit1302.

In step S1402, the subject area detection unit 1302 sets the face areacontained in the face area information as a subject area. In step S1403,the subject spatial information generation unit 1303 outputs subjectarea information calculated by the subject area detection unit 1302 asspatial information.

Note that the face detection unit 1301 may also output, as the face areainformation, the information of the direction of a face to the subjectarea detection unit 1302. The subject area detection unit 1302 thensets, based on the information of the input direction of the face, onlya face facing front as a subject area, for example, and a face inprofile does not need to be set as a subject area.

In the above description, an example for detecting a face area wasdescribed, but the present invention is not limited to detection of theface area. For example, a human body area may be detected in order touse human body detection information, and a general subject may bedetected. That is, it is sufficient that an area of a subject isdetected using a subject detection and recognition technique.

In the above description, an example of setting the face area as thesubject area was described, but the subject area detection unit 1302 mayset, as the subject area, an area encompassing the face area and thehuman body area.

Moreover, the subject area detection unit 1302 may set, as a subjectinner area, an area that has the same phase difference as the face areabased on phase difference information and face area informationcalculated during the AF control by the system control unit 50 describedin the first embodiment. The above has described the spatial informationcalculation unit 115 according to the third embodiment. Thearea-of-interest detection unit 116 of the present embodiment has thesame configuration and operations as those of the first and secondembodiments, and therefore detailed description thereof is omitted.

FIGS. 16A and 16B are diagrams for describing a degree of visualsaliency calculated by the area-of-interest detection unit 116 in thethird embodiment. The area-of-interest detection unit 116 in the thirdembodiment has the same configuration and performs the same operationsas those of the first and the second embodiments, so that subject innerareas and subject outer areas are set as in FIG. 16A, based on thespatial information described with reference to FIGS. 15A and 15B. Here,the subject inner areas are a900, a902, a904, and a906 in FIG. 16A. Asubject outer area corresponding to the subject inner area a900 is anarea a901 in FIG. 16A excluding a900. Similarly, subject outer areascorresponding to the subject inner areas a902, a904, and a906 are,respectively, areas a903, a905, and a907 excluding a902, a904, and a906.

The area-of-interest detection unit 116 then calculates the degree ofvisual saliency as in FIG. 16B based on the difference in feature amountbetween the subject inner area and the subject outer area. The value ofthe degree of visual saliency of an area a1000 is large because thedifference in feature amount between the subject inner area and thesubject outer area is large. The value of the degree of visual saliencyof an area a1002 is moderate because the difference in feature amountbetween the subject inner area and the subject outer area is moderate.The values of the degrees of visual saliency of an area a1001 and anarea a1003 are small because the differences in feature amount betweenthe subject inner areas and the subject outer areas are small.

Because the degree of visual saliency calculated in this manner involvesa visually conspicuous area of interest, the degree of visual saliencyis used for priority setting for various processes. Examples of suchprocesses include performing the AF control preferentially on this areaof interest.

For example, in the case where multiple faces are present as in FIG.16A, the system control unit 50 performs the AF control based on theinformation of phase differences calculated for the AF control and theinformation of an area of interest detected by the area-of-interestdetection unit 116, so that a face corresponding to the area of interestis preferentially in focus. Specifically, in the examples of FIGS. 16Aand 16B, the AF control is performed with priority given so that amongfour faces, a face on the left with a large degree of visual saliency isin focus. By performing the AF control preferentially on the area ofinterest in this manner, it is possible to cause the attention subject(area of interest) to be in focus without blurring the attention subject(area of interest).

OTHER EMBODIMENTS

Embodiments of the present invention can also be realized by a computerof a system or apparatus that reads out and executes computer executableinstructions (e.g., one or more programs) recorded on a storage medium(which may also be referred to more fully as a ‘non-transitorycomputer-readable storage medium’) to perform the functions of one ormore of the above-described Embodiments and/or that includes one or morecircuits (e.g., application specific integrated circuit (ASIC)) forperforming the functions of one or more of the above-describedembodiments, and by a method performed by the computer of the system orapparatus by, for example, reading out and executing the computerexecutable instructions from the storage medium to perform the functionsof one or more of the above-described embodiments and/or controlling theone or more circuits to perform the functions of one or more of theabove-described embodiments. The computer may comprise one or moreprocessors (e.g., central processing unit (CPU), micro processing unit(MPU)) and may include a network of separate computers or separateprocessors to read out and execute the computer executable instructions.The computer executable instructions may be provided to the computer,for example, from a network or the storage medium. The storage mediummay include, for example, one or more of a hard disk, a random-accessmemory (RAM), a read only memory (ROM), a storage of distributedcomputing systems, an optical disk (such as a compact disc (CD), digitalversatile disc (DVD), or Blu-ray Disc (BD)™), a flash memory device, amemory card, and the like.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2014-202121, filed Sep. 30, 2014, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. An image processing apparatus comprising: aspatial information calculation unit configured to calculate spatialinformation of a subject, the spatial information being information ofan area in which the subject in an image is predicted to be present; afirst area setting unit configured to set a first area in the imagebased on the spatial information; a second area setting unit configuredto set a second area outside the first area; a first feature amountcalculation unit configured to calculate a first feature amount of thefirst area; a second feature amount calculation unit configured tocalculate a second feature amount of the second area, the second featureamount being a feature amount of the same type as the first featureamount; and an saliency calculation unit configured to calculate adegree of visual saliency of the subject based on a difference betweenthe first feature amount and the second feature amount.
 2. The imageprocessing apparatus according to claim 1, wherein the spatialinformation includes at least one of a position and a size of thesubject in the image.
 3. The image processing apparatus according toclaim 2, wherein the first area setting unit sets a size of the firstarea based on the size of the subject contained in the spatialinformation.
 4. The image processing apparatus according to claim 2,wherein the first area setting unit sets a position of the first areabased on the position of the subject contained in the spatialinformation.
 5. The image processing apparatus according to claim 1,further comprising: a search area determination unit configured todetermine a search area that is larger than the first area and is usedfor calculating the degree of visual saliency, wherein the saliencycalculation unit sets a plurality of first areas in the search area. 6.The image processing apparatus according to claim 5, wherein the searcharea determination unit determines the search area based on the spatialinformation.
 7. The image processing apparatus according to claim 1,further comprising: a focus information calculation unit configured tocalculate information of a focused area, wherein the spatial informationcalculation unit calculates the spatial information of the subject basedon the information of the focused area.
 8. The image processingapparatus according to claim 7, wherein the information of the focusedarea is calculated based on information indicating whether or not thearea is in focus during a focusing operation.
 9. The image processingapparatus according to claim 1, further comprising: a moving bodyinformation calculation unit configured to calculate information of anarea in which a moving body is present, wherein the spatial informationcalculation unit calculates the spatial information of the subject basedon the information of the area in which the moving body is present. 10.The image processing apparatus according to claim 9, wherein theinformation of the area in which the moving body is present iscalculated based on an absolute difference between a plurality ofimages.
 11. The image processing apparatus according to claim 1, furthercomprising: a subject detection unit configured to calculate informationof an area in which a subject is present, wherein the spatialinformation calculation unit calculates spatial information of thesubject based on the information of the area in which the subject ispresent.
 12. The image processing apparatus according to claim 11,wherein the information of the area in which the subject is present iscalculated in accordance with at least one of face detection informationand human body detection information.
 13. The image processing apparatusaccording to claim 1, wherein priority for a target area for processingin an image is set based on the degree of visual saliency.
 14. The imageprocessing apparatus according to claim 1, wherein the saliencycalculation unit calculates a degree of visual saliency of only an areanear a center of a screen.
 15. The image processing apparatus accordingto claim 1, wherein the saliency calculation unit calculates degree ofvisual saliency of only an area whose area measurement is larger than apredetermined value.
 16. The image processing apparatus according toclaim 1, wherein the first and second feature amounts include at leastone of brightness, color, edge intensity, a brightness histogram, acolor histogram, and an edge intensity histogram.
 17. An imageprocessing method comprising: calculating spatial information of asubject, the spatial information being information of an area in which asubject in an image is predicted to be present; setting a first area inthe image based on the spatial information; setting a second areaoutside the first area; calculating a first feature amount of the firstarea; calculating a second feature amount of the second area, the secondfeature amount being a feature amount of the same type as the firstfeature amount; and calculating a degree of visual saliency of thesubject based on a difference between the first feature amount and thesecond feature amount.
 18. A non-transitory computer readable storagemedium storing a computer-executable program for executing a method forcontrolling an image processing apparatus using the computer, the methodcomprising: calculating spatial information of a subject, the spatialinformation being information of an area in which the subject in animage is predicted to be present; setting a first area in the imagebased on the spatial information; setting a second area outside thefirst area; calculating a first feature amount of the first area;calculating a second feature amount of the second area, the secondfeature amount being a feature amount of the same type as the firstfeature amount; and calculating a degree of visual saliency of thesubject based on a difference between the first feature amount and thesecond feature amount.